Analyzing Shakespeare¶
1.) To get a first idea, run a quick analysis on the text using Unix tools such as wc and grep to answer the following questions:¶
a: wc Shakespeare.txt: 124456 bytes, 901325 lines, 5458199 words
b: grep -c -i "by William Shakespeare" Shakespeare.txt: 38
2.)¶
The first part of the pipeline searches all occurences of "by William Shakespeare" in Shakespeare.txt. The Parameter -B 6 further includes the 6 lines before the selected pattern. The output of the first part is then matched again with a expression in three pieces:
The regex ^$ is denied by the option -e so all empty lines are excluded. The regex tr '\n' ' ' is used to replace all newlines with a space. The regex sed 's/ -- /\n/g' is used to replace all double dashes with a newline.
The execution involes 4 processes.
3.)¶
a) monotonically_increasing_id => filter b) filter(... isin copyright) c) withColumn when contains "by William Shakespeare" lag... d+e) repartition => groupBy => agg
